Chest X-Ray (Pneumonia): Image Classification w/Convolutional Neural Networks and Transfer Learning¶
Introduction¶
Pneumonia is a severe inflammatory condition of the lungs, primarily affecting the alveoli, and is a leading cause of death worldwide, especially in young children. Early and accurate diagnosis is crucial for effective treatment. A common diagnostic tool is the chest X-ray. However, visual interpretation of X-rays can be challenging, time-consuming, and prone to human error. This project explores the use of Convolutional Neural Networks (CNNs) and transfer learning to automate the detection of pneumonia from chest X-ray images, aiming to provide a rapid and objective screening tool to assist radiologists and clinicians.
Kaggle Dataset Link: https://www.kaggle.com/datasets/paultimothymooney/chest-xray-pneumonia/data
Dataset Information¶
The dataset contains 5,856 validated Chest X-Ray images. The images are split into a training set and a testing set of independent patients. Images are labeled as (disease:NORMAL/BACTERIA/VIRUS)-(randomized patient ID)-(image number of a patient).
Chest X-ray images (anterior-posterior) were selected from retrospective cohorts of pediatric patients of one to five years old from Guangzhou Women and Children’s Medical Center, Guangzhou. All chest X-ray imaging was performed as part of patients’ routine clinical care.
For the analysis of chest x-ray images, all chest radiographs were initially screened for quality control by removing all low quality or unreadable scans. The diagnoses for the images were then graded by two expert physicians before being cleared for training the AI system. In order to account for any grading errors, the evaluation set was also checked by a third expert.
The dataset is organised organized into training and testing folders. Training consist of 5232 images while the testing consist of 624 images.
Importing Packages and Dataset¶
import pandas as pd
import matplotlib as mlp
import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns
%matplotlib inline
pd.options.display.max_colwidth = 100
import random
import os
from IPython.display import Image, display
import matplotlib.cm as cm
from numpy.random import seed
seed(42)
random.seed(42)
os.environ['PYTHONHASHSEED'] = str(42)
os.environ['TF_DETERMINISTIC_OPS'] = '1'
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras import callbacks
from tensorflow.keras.models import Model
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.layers import Input
from tensorflow.keras.applications.vgg16 import preprocess_input as vgg_preprocess
import glob
import cv2
from collections import Counter
from tensorflow.random import set_seed
set_seed(42)
import warnings
warnings.filterwarnings('ignore')
from sklearn.metrics import confusion_matrix,\
roc_auc_score, ConfusionMatrixDisplay, roc_curve
# Checking if TensorFlow is in GPU mode
gpus = tf.config.experimental.list_physical_devices('GPU')
if gpus:
try:
# Currently, memory growth needs to be the same across GPUs
for gpu in gpus:
print(gpu)
tf.config.experimental.set_memory_growth(gpu, True)
print("TensorFlow is using the GPU.")
except RuntimeError as e:
print(e)
else:
print("TensorFlow is not using the GPU. Check your TensorFlow installation.")
PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU') TensorFlow is using the GPU.
Data¶
IMG_SIZE = 224
BATCH = 32
SEED = 42
main_path = "../Data/chest_xray"
train_path = os.path.join(main_path,"train")
test_path=os.path.join(main_path,"test")
train_normal = glob.glob(train_path+"/NORMAL/*.jpeg")
train_pneumonia = glob.glob(train_path+"/PNEUMONIA/*.jpeg")
test_normal = glob.glob(test_path+"/NORMAL/*.jpeg")
test_pneumonia = glob.glob(test_path+"/PNEUMONIA/*.jpeg")
train_list = [x for x in train_normal]
train_list.extend([x for x in train_pneumonia])
df_train = pd.DataFrame(np.concatenate([['Normal']*len(train_normal) , ['Pneumonia']*len(train_pneumonia)]), columns = ['class'])
df_train['image'] = [x for x in train_list]
test_list = [x for x in test_normal]
test_list.extend([x for x in test_pneumonia])
df_test = pd.DataFrame(np.concatenate([['Normal']*len(test_normal) , ['Pneumonia']*len(test_pneumonia)]), columns = ['class'])
df_test['image'] = [x for x in test_list]
df_train
| class | image | |
|---|---|---|
| 0 | Normal | ../Data/chest_xray/train/NORMAL/NORMAL2-IM-0927-0001.jpeg |
| 1 | Normal | ../Data/chest_xray/train/NORMAL/NORMAL2-IM-1056-0001.jpeg |
| 2 | Normal | ../Data/chest_xray/train/NORMAL/IM-0427-0001.jpeg |
| 3 | Normal | ../Data/chest_xray/train/NORMAL/NORMAL2-IM-1260-0001.jpeg |
| 4 | Normal | ../Data/chest_xray/train/NORMAL/IM-0656-0001-0001.jpeg |
| ... | ... | ... |
| 5227 | Pneumonia | ../Data/chest_xray/train/PNEUMONIA/person142_virus_288.jpeg |
| 5228 | Pneumonia | ../Data/chest_xray/train/PNEUMONIA/person364_bacteria_1659.jpeg |
| 5229 | Pneumonia | ../Data/chest_xray/train/PNEUMONIA/person1323_virus_2283.jpeg |
| 5230 | Pneumonia | ../Data/chest_xray/train/PNEUMONIA/person772_virus_1401.jpeg |
| 5231 | Pneumonia | ../Data/chest_xray/train/PNEUMONIA/person501_virus_1010.jpeg |
5232 rows × 2 columns
df_test
| class | image | |
|---|---|---|
| 0 | Normal | ../Data/chest_xray/test/NORMAL/IM-0031-0001.jpeg |
| 1 | Normal | ../Data/chest_xray/test/NORMAL/IM-0025-0001.jpeg |
| 2 | Normal | ../Data/chest_xray/test/NORMAL/NORMAL2-IM-0272-0001.jpeg |
| 3 | Normal | ../Data/chest_xray/test/NORMAL/NORMAL2-IM-0102-0001.jpeg |
| 4 | Normal | ../Data/chest_xray/test/NORMAL/NORMAL2-IM-0229-0001.jpeg |
| ... | ... | ... |
| 619 | Pneumonia | ../Data/chest_xray/test/PNEUMONIA/person120_bacteria_572.jpeg |
| 620 | Pneumonia | ../Data/chest_xray/test/PNEUMONIA/person171_bacteria_826.jpeg |
| 621 | Pneumonia | ../Data/chest_xray/test/PNEUMONIA/person109_bacteria_512.jpeg |
| 622 | Pneumonia | ../Data/chest_xray/test/PNEUMONIA/person83_bacteria_410.jpeg |
| 623 | Pneumonia | ../Data/chest_xray/test/PNEUMONIA/person112_bacteria_538.jpeg |
624 rows × 2 columns
Data Exploration¶
plt.figure(figsize=(4,4))
ax = sns.countplot(x='class', data=df_train, palette="mako")
plt.xlabel("Class", fontsize= 12)
plt.ylabel("# of Samples", fontsize= 12)
plt.ylim(0,5000)
plt.title('Train set')
plt.xticks([0,1], ['Normal', 'Pneumonia'], fontsize = 11)
for p in ax.patches:
ax.annotate((p.get_height()), (p.get_x()+0.30, p.get_height()+300), fontsize = 13)
plt.show()
plt.figure(figsize=(5,5))
df_train['class'].value_counts().plot(kind='pie',labels = ['',''], autopct='%1.1f%%', explode = [0,0.05])
plt.title('Train set')
plt.legend(labels=['Pneumonia', 'Normal'])
plt.show()
plt.figure(figsize=(6,4))
ax = sns.countplot(x='class', data=df_test, palette="mako")
plt.xlabel("Class", fontsize= 12)
plt.ylabel("# of Samples", fontsize= 12)
plt.ylim(0,500)
plt.title('Test set')
plt.xticks([0,1], ['Normal', 'Pneumonia'], fontsize = 11)
for p in ax.patches:
ax.annotate((p.get_height()), (p.get_x()+0.32, p.get_height()+20), fontsize = 13)
plt.show()
plt.figure(figsize=(7,5))
df_test['class'].value_counts().plot(kind='pie',labels = ['',''], autopct='%1.1f%%', explode = [0,0.05])
plt.title('Test set')
plt.legend(labels=['Pneumonia', 'Normal'])
plt.show()
The distributions from these datasets are a little different from each other. Both are slightly imbalanced, having more samples from the positive class (Pneumonia), with the training set being a little more imbalanced.
Before we move on to the next section, we will take a look at a few examples from each dataset.
print('Train Set - Normal')
plt.figure(figsize=(12,12))
for i in range(0, 12):
plt.subplot(3,4,i + 1)
img = cv2.imread(train_normal[i])
img = cv2.resize(img, (IMG_SIZE,IMG_SIZE))
plt.imshow(img)
plt.axis("off")
plt.tight_layout()
plt.show()
Train Set - Normal
print('Train Set - Pneumonia')
plt.figure(figsize=(12,12))
for i in range(0, 12):
plt.subplot(3,4,i + 1)
img = cv2.imread(train_pneumonia[i])
img = cv2.resize(img, (IMG_SIZE,IMG_SIZE))
plt.imshow(img)
plt.axis("off")
plt.tight_layout()
plt.show()
Train Set - Pneumonia
print('Test Set - Normal')
plt.figure(figsize=(12,12))
for i in range(0, 12):
plt.subplot(3,4,i + 1)
img = cv2.imread(test_normal[i])
img = cv2.resize(img, (IMG_SIZE,IMG_SIZE))
plt.imshow(img)
plt.axis("off")
plt.tight_layout()
plt.show()
Test Set - Normal
print('Test Set - Pneumonia')
plt.figure(figsize=(12,12))
for i in range(0, 12):
plt.subplot(3,4,i + 1)
img = cv2.imread(test_pneumonia[i])
img = cv2.resize(img, (IMG_SIZE,IMG_SIZE))
plt.imshow(img)
plt.axis("off")
plt.tight_layout()
plt.show()
Test Set - Pneumonia
EDA Conclusion¶
Data exploration revealed a significant class imbalance, with a much higher number of pneumonia cases than normal cases in the training set. This imbalance is a critical issue that can lead to a model that is biased towards the majority class. To address this, the data was balanced through data augmentation.
Data Preparation¶
First, we need to create a validation set. To do that, we apply a simple stratified split on the original train dataset, using 80% for actual training and 20% for validation purposes.
df_train
| class | image | |
|---|---|---|
| 0 | Normal | ../Data/chest_xray/train/NORMAL/NORMAL2-IM-0927-0001.jpeg |
| 1 | Normal | ../Data/chest_xray/train/NORMAL/NORMAL2-IM-1056-0001.jpeg |
| 2 | Normal | ../Data/chest_xray/train/NORMAL/IM-0427-0001.jpeg |
| 3 | Normal | ../Data/chest_xray/train/NORMAL/NORMAL2-IM-1260-0001.jpeg |
| 4 | Normal | ../Data/chest_xray/train/NORMAL/IM-0656-0001-0001.jpeg |
| ... | ... | ... |
| 5227 | Pneumonia | ../Data/chest_xray/train/PNEUMONIA/person142_virus_288.jpeg |
| 5228 | Pneumonia | ../Data/chest_xray/train/PNEUMONIA/person364_bacteria_1659.jpeg |
| 5229 | Pneumonia | ../Data/chest_xray/train/PNEUMONIA/person1323_virus_2283.jpeg |
| 5230 | Pneumonia | ../Data/chest_xray/train/PNEUMONIA/person772_virus_1401.jpeg |
| 5231 | Pneumonia | ../Data/chest_xray/train/PNEUMONIA/person501_virus_1010.jpeg |
5232 rows × 2 columns
# Separate by class
class1 = df_train[df_train['class'] == 'Normal']
class2 = df_train[df_train['class'] == 'Pneumonia']
# Pick equal numbers from each class for validation (based on smaller class size)
val_size = min(len(class1), len(class2)) // 2 # or set manually
val_class1 = class1.sample(val_size, random_state=SEED)
val_class2 = class2.sample(val_size, random_state=SEED)
# Combine into validation set
val_df = pd.concat([val_class1, val_class2])
# Remaining samples go to training
train_df = df_train.drop(val_df.index)
print("Class distribution in Training:")
print(train_df['class'].value_counts(normalize=True))
print()
print("Class distribution in Validation:")
print(val_df['class'].value_counts(normalize=True))
Class distribution in Training: class Pneumonia 0.82621 Normal 0.17379 Name: proportion, dtype: float64 Class distribution in Validation: class Normal 0.5 Pneumonia 0.5 Name: proportion, dtype: float64
train_df
| class | image | |
|---|---|---|
| 0 | Normal | ../Data/chest_xray/train/NORMAL/NORMAL2-IM-0927-0001.jpeg |
| 1 | Normal | ../Data/chest_xray/train/NORMAL/NORMAL2-IM-1056-0001.jpeg |
| 4 | Normal | ../Data/chest_xray/train/NORMAL/IM-0656-0001-0001.jpeg |
| 7 | Normal | ../Data/chest_xray/train/NORMAL/IM-0757-0001.jpeg |
| 8 | Normal | ../Data/chest_xray/train/NORMAL/NORMAL2-IM-1326-0001.jpeg |
| ... | ... | ... |
| 5225 | Pneumonia | ../Data/chest_xray/train/PNEUMONIA/person66_bacteria_325.jpeg |
| 5227 | Pneumonia | ../Data/chest_xray/train/PNEUMONIA/person142_virus_288.jpeg |
| 5228 | Pneumonia | ../Data/chest_xray/train/PNEUMONIA/person364_bacteria_1659.jpeg |
| 5230 | Pneumonia | ../Data/chest_xray/train/PNEUMONIA/person772_virus_1401.jpeg |
| 5231 | Pneumonia | ../Data/chest_xray/train/PNEUMONIA/person501_virus_1010.jpeg |
3884 rows × 2 columns
Counter(train_df['class'])
Counter({'Pneumonia': 3209, 'Normal': 675})
val_df
| class | image | |
|---|---|---|
| 289 | Normal | ../Data/chest_xray/train/NORMAL/NORMAL2-IM-0832-0001-0002.jpeg |
| 1036 | Normal | ../Data/chest_xray/train/NORMAL/NORMAL2-IM-0522-0001.jpeg |
| 535 | Normal | ../Data/chest_xray/train/NORMAL/NORMAL2-IM-0853-0001.jpeg |
| 346 | Normal | ../Data/chest_xray/train/NORMAL/NORMAL2-IM-0981-0001.jpeg |
| 1075 | Normal | ../Data/chest_xray/train/NORMAL/NORMAL2-IM-1277-0001.jpeg |
| ... | ... | ... |
| 4315 | Pneumonia | ../Data/chest_xray/train/PNEUMONIA/person1179_bacteria_3127.jpeg |
| 2211 | Pneumonia | ../Data/chest_xray/train/PNEUMONIA/person719_bacteria_2621.jpeg |
| 4492 | Pneumonia | ../Data/chest_xray/train/PNEUMONIA/person683_bacteria_2578.jpeg |
| 3253 | Pneumonia | ../Data/chest_xray/train/PNEUMONIA/person1636_bacteria_4338.jpeg |
| 1410 | Pneumonia | ../Data/chest_xray/train/PNEUMONIA/person1062_bacteria_2996.jpeg |
1348 rows × 2 columns
Counter(val_df['class'])
Counter({'Normal': 674, 'Pneumonia': 674})
Counter(df_test['class'])
Counter({'Pneumonia': 390, 'Normal': 234})
Data Augmentation¶
To counter the class imbalance and improve model generalization, data augmentation techniques were applied to the training set. This process artificially increases the number of training images by applying random transformations, such as zooming, horizontal and vertical shifting, and flipping. This not only balances the dataset but also helps the model learn to recognize features regardless of minor variations in the image, reducing overfitting. The augmented data also included both normal and pneumonia images.
train_datagen = ImageDataGenerator(
rescale=1/255,
featurewise_center=False, # set input mean to 0 over the dataset
samplewise_center=False, # set each sample mean to 0
featurewise_std_normalization=False, # divide inputs by std of the dataset
samplewise_std_normalization=False, # divide each input by its std
zca_whitening=False, # apply ZCA whitening
rotation_range = 30, # randomly rotate images in the range (degrees, 0 to 180)
zoom_range = 0.2, # Randomly zoom image
width_shift_range=0.1, # randomly shift images horizontally (fraction of total width)
height_shift_range=0.1, # randomly shift images vertically (fraction of total height)
horizontal_flip = True, # randomly flip images
vertical_flip=False) # randomly flip images
val_datagen = ImageDataGenerator(rescale=1/255.)
ds_train = train_datagen.flow_from_dataframe(
train_df,
x_col = 'image',
y_col = 'class',
target_size = (IMG_SIZE, IMG_SIZE),
class_mode = 'binary',
batch_size = BATCH,
seed = SEED
)
ds_val = val_datagen.flow_from_dataframe(
val_df,
x_col = 'image',
y_col = 'class',
target_size = (IMG_SIZE, IMG_SIZE),
class_mode = 'binary',
batch_size = BATCH,
seed = SEED
)
ds_test = val_datagen.flow_from_dataframe(
df_test,
x_col = 'image',
y_col = 'class',
target_size = (IMG_SIZE, IMG_SIZE),
class_mode = 'binary',
batch_size = 1,
shuffle = False
)
Found 3884 validated image filenames belonging to 2 classes. Found 1348 validated image filenames belonging to 2 classes. Found 624 validated image filenames belonging to 2 classes.
Model Implementation¶
This study implements and compares two main approaches for image classification: a custom CNN model and a transfer learning approach using pre-trained models.
Custom CNN Model¶
A custom CNN is designed and built from scratch. The model architecture consisted of multiple convolutional layers with max-pooling, followed by fully connected layers. The convolutional layers are responsible for learning hierarchical features from the images, while the dense layers perform the final classification. This model serves as a baseline to compare against the more sophisticated transfer learning models.
Transfer Learning¶
Transfer learning leverages knowledge from models pre-trained on a massive, general-purpose dataset (like ImageNet) to solve a new, more specific problem. The rationale behind this approach is that features learned by these models (e.g., edges, textures) are universally useful for image recognition tasks. This project utilized two popular pre-trained models as feature extractors: ResNet and VGG.
ResNet (Residual Networks): The ResNet architecture is known for its use of "residual blocks" which help to train very deep neural networks by allowing the flow of information to skip layers, thus mitigating the vanishing gradient problem.
VGG: The VGG architecture is a deep CNN with a simple, uniform structure. It is composed of a series of convolutional layers followed by max-pooling layers. It is less complex than ResNet, but still highly effective.
In this implementation, the pre-trained models' convolutional bases are used as fixed feature extractors, and new, trainable classification layers were added on top. Fine-tuning is also explored by unfreezing and training the last few layers of the pre-trained models to adapt them more specifically to the chest X-ray dataset.
#Setting callbakcs
early_stopping = callbacks.EarlyStopping(
monitor='val_loss',
patience=5,
min_delta=1e-7,
restore_best_weights=True,
)
checkpoint_cb = callbacks.ModelCheckpoint(
"xray_model.h5",
save_best_only=True
)
plateau = callbacks.ReduceLROnPlateau(
monitor='val_loss',
factor = 0.2,
patience = 5,
min_delt = 1e-7,
cooldown = 0,
verbose = 1
)
# Custom CNN
def CNNModel():
#Input shape = [width, height, color channels]
inputs = layers.Input(shape=(IMG_SIZE, IMG_SIZE, 3))
# Block One
x = layers.Conv2D(filters=16, kernel_size=3, padding='valid')(inputs)
x = layers.BatchNormalization()(x)
x = layers.Activation('relu')(x)
x = layers.MaxPool2D()(x)
x = layers.Dropout(0.2)(x)
# Block Two
x = layers.Conv2D(filters=32, kernel_size=3, padding='valid')(x)
x = layers.BatchNormalization()(x)
x = layers.Activation('relu')(x)
x = layers.MaxPool2D()(x)
x = layers.Dropout(0.2)(x)
# Block Three
x = layers.Conv2D(filters=64, kernel_size=3, padding='valid')(x)
x = layers.Conv2D(filters=64, kernel_size=3, padding='valid')(x)
x = layers.BatchNormalization()(x)
x = layers.Activation('relu')(x)
x = layers.MaxPool2D()(x)
x = layers.Dropout(0.4)(x)
# Head
#x = layers.BatchNormalization()(x)
x = layers.Flatten()(x)
x = layers.Dense(64, activation='relu')(x)
x = layers.Dropout(0.5)(x)
#Final Layer (Output)
output = layers.Dense(1, activation='sigmoid')(x)
model = keras.Model(inputs=[inputs], outputs=output)
return model
keras.backend.clear_session()
model = CNNModel()
model.compile(
loss='binary_crossentropy',
optimizer = keras.optimizers.Adam(learning_rate=1e-5),
metrics=[
'binary_accuracy',
])
model.summary()
Model: "functional"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩ │ input_layer (InputLayer) │ (None, 224, 224, 3) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ conv2d (Conv2D) │ (None, 222, 222, 16) │ 448 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ batch_normalization │ (None, 222, 222, 16) │ 64 │ │ (BatchNormalization) │ │ │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ activation (Activation) │ (None, 222, 222, 16) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ max_pooling2d (MaxPooling2D) │ (None, 111, 111, 16) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dropout (Dropout) │ (None, 111, 111, 16) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ conv2d_1 (Conv2D) │ (None, 109, 109, 32) │ 4,640 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ batch_normalization_1 │ (None, 109, 109, 32) │ 128 │ │ (BatchNormalization) │ │ │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ activation_1 (Activation) │ (None, 109, 109, 32) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ max_pooling2d_1 (MaxPooling2D) │ (None, 54, 54, 32) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dropout_1 (Dropout) │ (None, 54, 54, 32) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ conv2d_2 (Conv2D) │ (None, 52, 52, 64) │ 18,496 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ conv2d_3 (Conv2D) │ (None, 50, 50, 64) │ 36,928 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ batch_normalization_2 │ (None, 50, 50, 64) │ 256 │ │ (BatchNormalization) │ │ │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ activation_2 (Activation) │ (None, 50, 50, 64) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ max_pooling2d_2 (MaxPooling2D) │ (None, 25, 25, 64) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dropout_2 (Dropout) │ (None, 25, 25, 64) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ flatten (Flatten) │ (None, 40000) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense (Dense) │ (None, 64) │ 2,560,064 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dropout_3 (Dropout) │ (None, 64) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_1 (Dense) │ (None, 1) │ 65 │ └─────────────────────────────────┴────────────────────────┴───────────────┘
Total params: 2,621,089 (10.00 MB)
Trainable params: 2,620,865 (10.00 MB)
Non-trainable params: 224 (896.00 B)
epoch = 30
history = model.fit(
ds_train,
batch_size = BATCH, epochs = epoch,
validation_data=ds_val,
callbacks=[checkpoint_cb],
steps_per_epoch=(len(train_df)//BATCH),
validation_steps=(len(val_df)//BATCH)
)
Epoch 1/30 121/121 ━━━━━━━━━━━━━━━━━━━━ 43s 342ms/step - binary_accuracy: 0.7173 - loss: 1.3270 - val_binary_accuracy: 0.4993 - val_loss: 1.1312 Epoch 2/30 121/121 ━━━━━━━━━━━━━━━━━━━━ 7s 54ms/step - binary_accuracy: 0.6875 - loss: 1.5286 - val_binary_accuracy: 0.5007 - val_loss: 1.1403 Epoch 3/30 121/121 ━━━━━━━━━━━━━━━━━━━━ 41s 338ms/step - binary_accuracy: 0.7400 - loss: 1.2232 - val_binary_accuracy: 0.5015 - val_loss: 2.0678 Epoch 4/30 121/121 ━━━━━━━━━━━━━━━━━━━━ 7s 54ms/step - binary_accuracy: 0.6250 - loss: 1.3407 - val_binary_accuracy: 0.5015 - val_loss: 2.1013 Epoch 5/30 121/121 ━━━━━━━━━━━━━━━━━━━━ 40s 333ms/step - binary_accuracy: 0.7702 - loss: 1.2087 - val_binary_accuracy: 0.5000 - val_loss: 2.7930 Epoch 6/30 121/121 ━━━━━━━━━━━━━━━━━━━━ 7s 56ms/step - binary_accuracy: 0.7188 - loss: 1.1368 - val_binary_accuracy: 0.5007 - val_loss: 2.8099 Epoch 7/30 121/121 ━━━━━━━━━━━━━━━━━━━━ 41s 340ms/step - binary_accuracy: 0.7937 - loss: 1.1036 - val_binary_accuracy: 0.5000 - val_loss: 3.3434 Epoch 8/30 121/121 ━━━━━━━━━━━━━━━━━━━━ 7s 55ms/step - binary_accuracy: 0.8438 - loss: 1.0557 - val_binary_accuracy: 0.5007 - val_loss: 3.3585 Epoch 9/30 121/121 ━━━━━━━━━━━━━━━━━━━━ 41s 342ms/step - binary_accuracy: 0.8068 - loss: 1.0610 - val_binary_accuracy: 0.4985 - val_loss: 3.0004 Epoch 10/30 121/121 ━━━━━━━━━━━━━━━━━━━━ 7s 55ms/step - binary_accuracy: 0.8438 - loss: 1.0664 - val_binary_accuracy: 0.5000 - val_loss: 2.9628 Epoch 11/30 121/121 ━━━━━━━━━━━━━━━━━━━━ 41s 336ms/step - binary_accuracy: 0.8300 - loss: 0.9987 - val_binary_accuracy: 0.5320 - val_loss: 1.8924 Epoch 12/30 121/121 ━━━━━━━━━━━━━━━━━━━━ 7s 54ms/step - binary_accuracy: 0.8438 - loss: 1.0722 - val_binary_accuracy: 0.5312 - val_loss: 1.8967 Epoch 13/30 121/121 ━━━━━━━━━━━━━━━━━━━━ 0s 277ms/step - binary_accuracy: 0.8281 - loss: 0.9824
WARNING:absl:You are saving your model as an HDF5 file via `model.save()` or `keras.saving.save_model(model)`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')` or `keras.saving.save_model(model, 'my_model.keras')`.
121/121 ━━━━━━━━━━━━━━━━━━━━ 40s 334ms/step - binary_accuracy: 0.8282 - loss: 0.9823 - val_binary_accuracy: 0.7225 - val_loss: 0.7594 Epoch 14/30 121/121 ━━━━━━━━━━━━━━━━━━━━ 7s 55ms/step - binary_accuracy: 0.8125 - loss: 0.6716 - val_binary_accuracy: 0.7076 - val_loss: 0.8022 Epoch 15/30 121/121 ━━━━━━━━━━━━━━━━━━━━ 0s 275ms/step - binary_accuracy: 0.8384 - loss: 0.9487
WARNING:absl:You are saving your model as an HDF5 file via `model.save()` or `keras.saving.save_model(model)`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')` or `keras.saving.save_model(model, 'my_model.keras')`.
121/121 ━━━━━━━━━━━━━━━━━━━━ 40s 331ms/step - binary_accuracy: 0.8384 - loss: 0.9484 - val_binary_accuracy: 0.7991 - val_loss: 0.5577 Epoch 16/30 1/121 ━━━━━━━━━━━━━━━━━━━━ 35s 295ms/step - binary_accuracy: 0.8438 - loss: 1.1030
WARNING:absl:You are saving your model as an HDF5 file via `model.save()` or `keras.saving.save_model(model)`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')` or `keras.saving.save_model(model, 'my_model.keras')`.
121/121 ━━━━━━━━━━━━━━━━━━━━ 7s 54ms/step - binary_accuracy: 0.8438 - loss: 1.1030 - val_binary_accuracy: 0.8036 - val_loss: 0.5548 Epoch 17/30 121/121 ━━━━━━━━━━━━━━━━━━━━ 40s 333ms/step - binary_accuracy: 0.8382 - loss: 0.9369 - val_binary_accuracy: 0.7917 - val_loss: 0.6209 Epoch 18/30 121/121 ━━━━━━━━━━━━━━━━━━━━ 7s 53ms/step - binary_accuracy: 0.8750 - loss: 0.5270 - val_binary_accuracy: 0.7924 - val_loss: 0.6147 Epoch 19/30 121/121 ━━━━━━━━━━━━━━━━━━━━ 0s 274ms/step - binary_accuracy: 0.8462 - loss: 0.9651
WARNING:absl:You are saving your model as an HDF5 file via `model.save()` or `keras.saving.save_model(model)`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')` or `keras.saving.save_model(model, 'my_model.keras')`.
121/121 ━━━━━━━━━━━━━━━━━━━━ 40s 329ms/step - binary_accuracy: 0.8462 - loss: 0.9651 - val_binary_accuracy: 0.8378 - val_loss: 0.5024 Epoch 20/30 1/121 ━━━━━━━━━━━━━━━━━━━━ 34s 285ms/step - binary_accuracy: 0.9062 - loss: 0.5138
WARNING:absl:You are saving your model as an HDF5 file via `model.save()` or `keras.saving.save_model(model)`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')` or `keras.saving.save_model(model, 'my_model.keras')`.
121/121 ━━━━━━━━━━━━━━━━━━━━ 7s 55ms/step - binary_accuracy: 0.9062 - loss: 0.5138 - val_binary_accuracy: 0.8460 - val_loss: 0.4784 Epoch 21/30 121/121 ━━━━━━━━━━━━━━━━━━━━ 40s 329ms/step - binary_accuracy: 0.8429 - loss: 1.0802 - val_binary_accuracy: 0.8318 - val_loss: 0.5476 Epoch 22/30 121/121 ━━━━━━━━━━━━━━━━━━━━ 7s 54ms/step - binary_accuracy: 0.9688 - loss: 0.0644 - val_binary_accuracy: 0.8266 - val_loss: 0.5725 Epoch 23/30 121/121 ━━━━━━━━━━━━━━━━━━━━ 0s 275ms/step - binary_accuracy: 0.8385 - loss: 1.1167
WARNING:absl:You are saving your model as an HDF5 file via `model.save()` or `keras.saving.save_model(model)`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')` or `keras.saving.save_model(model, 'my_model.keras')`.
121/121 ━━━━━━━━━━━━━━━━━━━━ 40s 332ms/step - binary_accuracy: 0.8386 - loss: 1.1158 - val_binary_accuracy: 0.8862 - val_loss: 0.3832 Epoch 24/30 121/121 ━━━━━━━━━━━━━━━━━━━━ 7s 54ms/step - binary_accuracy: 0.9062 - loss: 0.2684 - val_binary_accuracy: 0.8802 - val_loss: 0.3967 Epoch 25/30 121/121 ━━━━━━━━━━━━━━━━━━━━ 40s 328ms/step - binary_accuracy: 0.8535 - loss: 0.9583 - val_binary_accuracy: 0.8728 - val_loss: 0.4302 Epoch 26/30 121/121 ━━━━━━━━━━━━━━━━━━━━ 7s 56ms/step - binary_accuracy: 0.7812 - loss: 1.1075 - val_binary_accuracy: 0.8720 - val_loss: 0.4279 Epoch 27/30 121/121 ━━━━━━━━━━━━━━━━━━━━ 0s 274ms/step - binary_accuracy: 0.8513 - loss: 1.0832
WARNING:absl:You are saving your model as an HDF5 file via `model.save()` or `keras.saving.save_model(model)`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')` or `keras.saving.save_model(model, 'my_model.keras')`.
121/121 ━━━━━━━━━━━━━━━━━━━━ 40s 330ms/step - binary_accuracy: 0.8513 - loss: 1.0831 - val_binary_accuracy: 0.9226 - val_loss: 0.3304 Epoch 28/30 1/121 ━━━━━━━━━━━━━━━━━━━━ 35s 298ms/step - binary_accuracy: 0.8750 - loss: 0.5485
WARNING:absl:You are saving your model as an HDF5 file via `model.save()` or `keras.saving.save_model(model)`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')` or `keras.saving.save_model(model, 'my_model.keras')`.
121/121 ━━━━━━━━━━━━━━━━━━━━ 7s 55ms/step - binary_accuracy: 0.8750 - loss: 0.5485 - val_binary_accuracy: 0.9234 - val_loss: 0.3296 Epoch 29/30 121/121 ━━━━━━━━━━━━━━━━━━━━ 0s 273ms/step - binary_accuracy: 0.8659 - loss: 0.8973
WARNING:absl:You are saving your model as an HDF5 file via `model.save()` or `keras.saving.save_model(model)`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')` or `keras.saving.save_model(model, 'my_model.keras')`.
121/121 ━━━━━━━━━━━━━━━━━━━━ 40s 329ms/step - binary_accuracy: 0.8659 - loss: 0.8978 - val_binary_accuracy: 0.9226 - val_loss: 0.3292 Epoch 30/30 121/121 ━━━━━━━━━━━━━━━━━━━━ 7s 55ms/step - binary_accuracy: 0.7812 - loss: 0.9056 - val_binary_accuracy: 0.9234 - val_loss: 0.3303
def plot_loss_acc(history, model_name):
acc = history.history['binary_accuracy']
val_acc = history.history['val_binary_accuracy']
loss = history.history['loss']
val_loss = history.history['val_loss']
epochs_range = history.epoch
plt.figure(figsize=(7, 4))
plt.subplot(1, 2, 1)
plt.plot(epochs_range, acc, label='Training Accuracy')
plt.plot(epochs_range, val_acc, label='Validation Accuracy')
plt.legend(loc='lower right')
plt.title('Training and Validation Accuracy')
plt.subplot(1, 2, 2)
plt.plot(epochs_range, loss, label='Training Loss')
plt.plot(epochs_range, val_loss, label='Validation Loss')
plt.legend(loc='upper right')
plt.title('Training and Validation Loss')
plt.suptitle(f'Model Training Performance: {model_name}', fontsize=14)
plt.show()
plot_loss_acc(history, "CNN")
score = model.evaluate(ds_val, steps = len(val_df)//BATCH, verbose = 0)
print('Val loss:', score[0])
print('Val accuracy:', score[1])
Val loss: 0.3251897096633911 Val accuracy: 0.9233630895614624
score = model.evaluate(ds_test, steps = len(df_test), verbose = 0)
print('Test loss:', score[0])
print('Test accuracy:', score[1])
Test loss: 0.8861863613128662 Test accuracy: 0.8028846383094788
Performance Metrics¶
def evaluate_model(model, model_name):
print(f'Model: {model_name}')
# Predictions
y_pred_proba = model.predict(ds_test, steps=len(df_test), verbose=1)
y_pred = (y_pred_proba > 0.5).astype("int32").ravel()
y_true = ds_test.classes
# Confusion Matrix + Metrics
cm = confusion_matrix(y_true, y_pred)
TN, FP, FN, TP = cm.ravel()
accuracy = (TP+TN) / (TP+TN+FN+FP)
sensitivity = TP / (TP + FN)
specificity = TN / (TN + FP)
precision = TP / (TP + FP)
f1_score = 2*precision*sensitivity / (precision + sensitivity)
print(f'Accuracy: {accuracy:.4f}')
print(f"Precision: {precision:.4f}")
print(f"Sensitivity (Recall): {sensitivity:.4f}")
print(f"Specificity: {specificity:.4f}")
print(f"F1 Score: {f1_score:.4f}")
# ROC values
fpr, tpr, _ = roc_curve(y_true, y_pred_proba)
roc_auc = roc_auc_score(y_true, y_pred_proba)
# Side-by-side plots
_, axes = plt.subplots(1, 2, figsize=(10, 5))
# Confusion matrix
disp = ConfusionMatrixDisplay(confusion_matrix=cm)
disp.plot(cmap="Blues", ax=axes[0], colorbar=False)
axes[0].set_title("Confusion Matrix")
# ROC curve
axes[1].plot(fpr, tpr, label=f"AUC = {roc_auc:.4f}")
axes[1].plot([0, 1], [0, 1], 'k--')
axes[1].set_xlabel("False Positive Rate")
axes[1].set_ylabel("True Positive Rate")
axes[1].set_title("ROC Curve")
axes[1].legend(loc="lower right")
plt.suptitle(f"Evaluation Results: {model_name}", fontsize=14)
plt.tight_layout()
plt.show()
# Performance Metrics
evaluate_model(model, 'CNN')
Model: CNN 624/624 ━━━━━━━━━━━━━━━━━━━━ 4s 6ms/step Accuracy: 0.8029 Precision: 0.7811 Sensitivity (Recall): 0.9513 Specificity: 0.5556 F1 Score: 0.8578
Transfer Learning¶
ResNet¶
# ResNet152V2 model
base_model_resnet = tf.keras.applications.ResNet152V2(
weights='imagenet',
input_shape=(IMG_SIZE, IMG_SIZE, 3),
include_top=False)
base_model_resnet.trainable = False
def ResNetModel():
#Input shape = [width, height, color channels]
inputs = layers.Input(shape=(IMG_SIZE, IMG_SIZE, 3))
x = base_model_resnet(inputs)
# Head
x = layers.GlobalAveragePooling2D()(x)
x = layers.Dense(128, activation='relu')(x)
x = layers.Dropout(0.1)(x)
#Final Layer (Output)
output = layers.Dense(1, activation='sigmoid')(x)
model = keras.Model(inputs=[inputs], outputs=output)
return model
keras.backend.clear_session()
model_resnet = ResNetModel()
model_resnet.compile(
loss='binary_crossentropy',
optimizer = keras.optimizers.Adam(learning_rate=1e-5),
metrics=['binary_accuracy']
)
model_resnet.summary()
Model: "functional"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩ │ input_layer (InputLayer) │ (None, 224, 224, 3) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ resnet152v2 (Functional) │ (None, 7, 7, 2048) │ 58,331,648 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ global_average_pooling2d │ (None, 2048) │ 0 │ │ (GlobalAveragePooling2D) │ │ │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense (Dense) │ (None, 128) │ 262,272 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dropout (Dropout) │ (None, 128) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_1 (Dense) │ (None, 1) │ 129 │ └─────────────────────────────────┴────────────────────────┴───────────────┘
Total params: 58,594,049 (223.52 MB)
Trainable params: 262,401 (1.00 MB)
Non-trainable params: 58,331,648 (222.52 MB)
checkpoint_resnet = callbacks.ModelCheckpoint(
"xray_resnet_model.h5",
save_best_only=True
)
resnet_epoch = 15
history_resnet = model_resnet.fit(
ds_train,
batch_size = BATCH, epochs = resnet_epoch,
validation_data=ds_val,
callbacks=[checkpoint_resnet, early_stopping],
steps_per_epoch=(len(train_df)//BATCH),
validation_steps=(len(val_df)//BATCH)
)
Epoch 1/15 121/121 ━━━━━━━━━━━━━━━━━━━━ 0s 529ms/step - binary_accuracy: 0.7718 - loss: 0.5440
WARNING:absl:You are saving your model as an HDF5 file via `model.save()` or `keras.saving.save_model(model)`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')` or `keras.saving.save_model(model, 'my_model.keras')`.
121/121 ━━━━━━━━━━━━━━━━━━━━ 90s 744ms/step - binary_accuracy: 0.7720 - loss: 0.5437 - val_binary_accuracy: 0.6183 - val_loss: 0.6832 Epoch 2/15 1/121 ━━━━━━━━━━━━━━━━━━━━ 1:13 609ms/step - binary_accuracy: 0.8750 - loss: 0.3647
WARNING:absl:You are saving your model as an HDF5 file via `model.save()` or `keras.saving.save_model(model)`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')` or `keras.saving.save_model(model, 'my_model.keras')`.
121/121 ━━━━━━━━━━━━━━━━━━━━ 20s 165ms/step - binary_accuracy: 0.8750 - loss: 0.3647 - val_binary_accuracy: 0.6183 - val_loss: 0.6815 Epoch 3/15 121/121 ━━━━━━━━━━━━━━━━━━━━ 0s 520ms/step - binary_accuracy: 0.8273 - loss: 0.4036
WARNING:absl:You are saving your model as an HDF5 file via `model.save()` or `keras.saving.save_model(model)`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')` or `keras.saving.save_model(model, 'my_model.keras')`.
121/121 ━━━━━━━━━━━━━━━━━━━━ 83s 687ms/step - binary_accuracy: 0.8274 - loss: 0.4035 - val_binary_accuracy: 0.7671 - val_loss: 0.4560 Epoch 4/15 121/121 ━━━━━━━━━━━━━━━━━━━━ 20s 162ms/step - binary_accuracy: 0.9062 - loss: 0.2225 - val_binary_accuracy: 0.7671 - val_loss: 0.4560 Epoch 5/15 121/121 ━━━━━━━━━━━━━━━━━━━━ 0s 510ms/step - binary_accuracy: 0.8502 - loss: 0.3232
WARNING:absl:You are saving your model as an HDF5 file via `model.save()` or `keras.saving.save_model(model)`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')` or `keras.saving.save_model(model, 'my_model.keras')`.
121/121 ━━━━━━━━━━━━━━━━━━━━ 82s 679ms/step - binary_accuracy: 0.8503 - loss: 0.3231 - val_binary_accuracy: 0.8222 - val_loss: 0.3780 Epoch 6/15 121/121 ━━━━━━━━━━━━━━━━━━━━ 20s 161ms/step - binary_accuracy: 0.7812 - loss: 0.3450 - val_binary_accuracy: 0.8207 - val_loss: 0.3783 Epoch 7/15 121/121 ━━━━━━━━━━━━━━━━━━━━ 0s 524ms/step - binary_accuracy: 0.8809 - loss: 0.2798
WARNING:absl:You are saving your model as an HDF5 file via `model.save()` or `keras.saving.save_model(model)`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')` or `keras.saving.save_model(model, 'my_model.keras')`.
121/121 ━━━━━━━━━━━━━━━━━━━━ 84s 697ms/step - binary_accuracy: 0.8809 - loss: 0.2798 - val_binary_accuracy: 0.8676 - val_loss: 0.3172 Epoch 8/15 121/121 ━━━━━━━━━━━━━━━━━━━━ 20s 162ms/step - binary_accuracy: 0.8125 - loss: 0.3968 - val_binary_accuracy: 0.8668 - val_loss: 0.3178 Epoch 9/15 121/121 ━━━━━━━━━━━━━━━━━━━━ 0s 510ms/step - binary_accuracy: 0.9065 - loss: 0.2448
WARNING:absl:You are saving your model as an HDF5 file via `model.save()` or `keras.saving.save_model(model)`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')` or `keras.saving.save_model(model, 'my_model.keras')`.
121/121 ━━━━━━━━━━━━━━━━━━━━ 82s 680ms/step - binary_accuracy: 0.9065 - loss: 0.2448 - val_binary_accuracy: 0.8884 - val_loss: 0.2777 Epoch 10/15 121/121 ━━━━━━━━━━━━━━━━━━━━ 20s 159ms/step - binary_accuracy: 0.7812 - loss: 0.3717 - val_binary_accuracy: 0.8884 - val_loss: 0.2793 Epoch 11/15 121/121 ━━━━━━━━━━━━━━━━━━━━ 0s 510ms/step - binary_accuracy: 0.9037 - loss: 0.2320
WARNING:absl:You are saving your model as an HDF5 file via `model.save()` or `keras.saving.save_model(model)`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')` or `keras.saving.save_model(model, 'my_model.keras')`.
121/121 ━━━━━━━━━━━━━━━━━━━━ 81s 674ms/step - binary_accuracy: 0.9037 - loss: 0.2319 - val_binary_accuracy: 0.8958 - val_loss: 0.2628 Epoch 12/15 1/121 ━━━━━━━━━━━━━━━━━━━━ 1:02 523ms/step - binary_accuracy: 0.8750 - loss: 0.2598
WARNING:absl:You are saving your model as an HDF5 file via `model.save()` or `keras.saving.save_model(model)`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')` or `keras.saving.save_model(model, 'my_model.keras')`.
121/121 ━━━━━━━━━━━━━━━━━━━━ 20s 164ms/step - binary_accuracy: 0.8750 - loss: 0.2598 - val_binary_accuracy: 0.8966 - val_loss: 0.2610 Epoch 13/15 121/121 ━━━━━━━━━━━━━━━━━━━━ 0s 503ms/step - binary_accuracy: 0.9182 - loss: 0.2001
WARNING:absl:You are saving your model as an HDF5 file via `model.save()` or `keras.saving.save_model(model)`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')` or `keras.saving.save_model(model, 'my_model.keras')`.
121/121 ━━━━━━━━━━━━━━━━━━━━ 81s 672ms/step - binary_accuracy: 0.9182 - loss: 0.2001 - val_binary_accuracy: 0.9070 - val_loss: 0.2378 Epoch 14/15 121/121 ━━━━━━━━━━━━━━━━━━━━ 20s 160ms/step - binary_accuracy: 0.9375 - loss: 0.1994 - val_binary_accuracy: 0.9070 - val_loss: 0.2381 Epoch 15/15 121/121 ━━━━━━━━━━━━━━━━━━━━ 0s 512ms/step - binary_accuracy: 0.9150 - loss: 0.1989
WARNING:absl:You are saving your model as an HDF5 file via `model.save()` or `keras.saving.save_model(model)`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')` or `keras.saving.save_model(model, 'my_model.keras')`.
121/121 ━━━━━━━━━━━━━━━━━━━━ 82s 680ms/step - binary_accuracy: 0.9150 - loss: 0.1990 - val_binary_accuracy: 0.9107 - val_loss: 0.2262
plot_loss_acc(history_resnet, 'ResNet')
score = model_resnet.evaluate(ds_test, steps = len(df_test), verbose = 0)
print('Test loss:', score[0])
print('Test accuracy:', score[1])
Test loss: 0.5046661496162415 Test accuracy: 0.7836538553237915
Performance Metrics¶
# Performance Metrics
evaluate_model(model_resnet, 'ResNet')
Model: ResNet 624/624 ━━━━━━━━━━━━━━━━━━━━ 33s 44ms/step Accuracy: 0.7837 Precision: 0.7576 Sensitivity (Recall): 0.9615 Specificity: 0.4872 F1 Score: 0.8475
Fine Tuning the ResNet Model¶
base_model_resnet_v2 = tf.keras.applications.ResNet152V2(
weights='imagenet',
input_shape=(IMG_SIZE, IMG_SIZE, 3),
include_top=False)
base_model_resnet_v2.trainable = True
# Freeze all layers except for the
for layer in base_model_resnet_v2.layers[:-13]:
layer.trainable = False
def ResNetModelV2():
#Input shape = [width, height, color channels]
inputs = layers.Input(shape=(IMG_SIZE, IMG_SIZE, 3))
x = base_model_resnet_v2(inputs)
# Head
x = layers.GlobalAveragePooling2D()(x)
x = layers.Dense(128, activation='relu')(x)
x = layers.Dropout(0.1)(x)
#Final Layer (Output)
output = layers.Dense(1, activation='sigmoid')(x)
model = keras.Model(inputs=[inputs], outputs=output)
return model
model_resnet_v2 = ResNetModelV2()
model_resnet_v2.compile(
loss='binary_crossentropy',
optimizer = keras.optimizers.Adam(learning_rate=1e-5),
metrics=['binary_accuracy']
)
model_resnet_v2.summary()
Model: "functional_1"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩ │ input_layer_2 (InputLayer) │ (None, 224, 224, 3) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ resnet152v2 (Functional) │ (None, 7, 7, 2048) │ 58,331,648 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ global_average_pooling2d_1 │ (None, 2048) │ 0 │ │ (GlobalAveragePooling2D) │ │ │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_2 (Dense) │ (None, 128) │ 262,272 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dropout_1 (Dropout) │ (None, 128) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_3 (Dense) │ (None, 1) │ 129 │ └─────────────────────────────────┴────────────────────────┴───────────────┘
Total params: 58,594,049 (223.52 MB)
Trainable params: 4,731,137 (18.05 MB)
Non-trainable params: 53,862,912 (205.47 MB)
checkpoint_resnet_v2 = callbacks.ModelCheckpoint(
"xray_resnet_v2_model.h5",
save_best_only=True
)
resnet_epoch = 15
history_resnet_v2 = model_resnet_v2.fit(
ds_train,
batch_size = BATCH, epochs = resnet_epoch,
validation_data=ds_val,
callbacks=[checkpoint_resnet_v2],
steps_per_epoch=(len(train_df)//BATCH),
validation_steps=(len(val_df)//BATCH)
)
Epoch 1/15 121/121 ━━━━━━━━━━━━━━━━━━━━ 0s 559ms/step - binary_accuracy: 0.7742 - loss: 0.5064
WARNING:absl:You are saving your model as an HDF5 file via `model.save()` or `keras.saving.save_model(model)`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')` or `keras.saving.save_model(model, 'my_model.keras')`.
121/121 ━━━━━━━━━━━━━━━━━━━━ 93s 772ms/step - binary_accuracy: 0.7746 - loss: 0.5056 - val_binary_accuracy: 0.8705 - val_loss: 0.3626 Epoch 2/15 1/121 ━━━━━━━━━━━━━━━━━━━━ 1:08 572ms/step - binary_accuracy: 1.0000 - loss: 0.2121
WARNING:absl:You are saving your model as an HDF5 file via `model.save()` or `keras.saving.save_model(model)`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')` or `keras.saving.save_model(model, 'my_model.keras')`.
121/121 ━━━━━━━━━━━━━━━━━━━━ 21s 170ms/step - binary_accuracy: 1.0000 - loss: 0.2121 - val_binary_accuracy: 0.8720 - val_loss: 0.3592 Epoch 3/15 121/121 ━━━━━━━━━━━━━━━━━━━━ 0s 529ms/step - binary_accuracy: 0.9045 - loss: 0.2558
WARNING:absl:You are saving your model as an HDF5 file via `model.save()` or `keras.saving.save_model(model)`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')` or `keras.saving.save_model(model, 'my_model.keras')`.
121/121 ━━━━━━━━━━━━━━━━━━━━ 84s 696ms/step - binary_accuracy: 0.9045 - loss: 0.2557 - val_binary_accuracy: 0.9263 - val_loss: 0.2390 Epoch 4/15 1/121 ━━━━━━━━━━━━━━━━━━━━ 1:10 589ms/step - binary_accuracy: 0.9062 - loss: 0.2135
WARNING:absl:You are saving your model as an HDF5 file via `model.save()` or `keras.saving.save_model(model)`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')` or `keras.saving.save_model(model, 'my_model.keras')`.
121/121 ━━━━━━━━━━━━━━━━━━━━ 21s 167ms/step - binary_accuracy: 0.9062 - loss: 0.2135 - val_binary_accuracy: 0.9263 - val_loss: 0.2388 Epoch 5/15 121/121 ━━━━━━━━━━━━━━━━━━━━ 0s 532ms/step - binary_accuracy: 0.9226 - loss: 0.1914
WARNING:absl:You are saving your model as an HDF5 file via `model.save()` or `keras.saving.save_model(model)`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')` or `keras.saving.save_model(model, 'my_model.keras')`.
121/121 ━━━━━━━━━━━━━━━━━━━━ 84s 699ms/step - binary_accuracy: 0.9227 - loss: 0.1913 - val_binary_accuracy: 0.9345 - val_loss: 0.1907 Epoch 6/15 1/121 ━━━━━━━━━━━━━━━━━━━━ 1:09 580ms/step - binary_accuracy: 0.9688 - loss: 0.1148
WARNING:absl:You are saving your model as an HDF5 file via `model.save()` or `keras.saving.save_model(model)`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')` or `keras.saving.save_model(model, 'my_model.keras')`.
121/121 ━━━━━━━━━━━━━━━━━━━━ 21s 168ms/step - binary_accuracy: 0.9688 - loss: 0.1148 - val_binary_accuracy: 0.9353 - val_loss: 0.1899 Epoch 7/15 121/121 ━━━━━━━━━━━━━━━━━━━━ 0s 531ms/step - binary_accuracy: 0.9410 - loss: 0.1587
WARNING:absl:You are saving your model as an HDF5 file via `model.save()` or `keras.saving.save_model(model)`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')` or `keras.saving.save_model(model, 'my_model.keras')`.
121/121 ━━━━━━━━━━━━━━━━━━━━ 85s 701ms/step - binary_accuracy: 0.9410 - loss: 0.1587 - val_binary_accuracy: 0.9449 - val_loss: 0.1555 Epoch 8/15 121/121 ━━━━━━━━━━━━━━━━━━━━ 21s 166ms/step - binary_accuracy: 0.9375 - loss: 0.1642 - val_binary_accuracy: 0.9449 - val_loss: 0.1566 Epoch 9/15 121/121 ━━━━━━━━━━━━━━━━━━━━ 0s 530ms/step - binary_accuracy: 0.9466 - loss: 0.1411
WARNING:absl:You are saving your model as an HDF5 file via `model.save()` or `keras.saving.save_model(model)`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')` or `keras.saving.save_model(model, 'my_model.keras')`.
121/121 ━━━━━━━━━━━━━━━━━━━━ 85s 702ms/step - binary_accuracy: 0.9465 - loss: 0.1412 - val_binary_accuracy: 0.9464 - val_loss: 0.1468 Epoch 10/15 121/121 ━━━━━━━━━━━━━━━━━━━━ 20s 164ms/step - binary_accuracy: 1.0000 - loss: 0.0772 - val_binary_accuracy: 0.9457 - val_loss: 0.1469 Epoch 11/15 121/121 ━━━━━━━━━━━━━━━━━━━━ 0s 539ms/step - binary_accuracy: 0.9508 - loss: 0.1381
WARNING:absl:You are saving your model as an HDF5 file via `model.save()` or `keras.saving.save_model(model)`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')` or `keras.saving.save_model(model, 'my_model.keras')`.
121/121 ━━━━━━━━━━━━━━━━━━━━ 86s 709ms/step - binary_accuracy: 0.9508 - loss: 0.1380 - val_binary_accuracy: 0.9472 - val_loss: 0.1424 Epoch 12/15 1/121 ━━━━━━━━━━━━━━━━━━━━ 1:08 575ms/step - binary_accuracy: 1.0000 - loss: 0.0725
WARNING:absl:You are saving your model as an HDF5 file via `model.save()` or `keras.saving.save_model(model)`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')` or `keras.saving.save_model(model, 'my_model.keras')`.
121/121 ━━━━━━━━━━━━━━━━━━━━ 21s 170ms/step - binary_accuracy: 1.0000 - loss: 0.0725 - val_binary_accuracy: 0.9464 - val_loss: 0.1419 Epoch 13/15 121/121 ━━━━━━━━━━━━━━━━━━━━ 0s 536ms/step - binary_accuracy: 0.9462 - loss: 0.1307
WARNING:absl:You are saving your model as an HDF5 file via `model.save()` or `keras.saving.save_model(model)`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')` or `keras.saving.save_model(model, 'my_model.keras')`.
121/121 ━━━━━━━━━━━━━━━━━━━━ 85s 702ms/step - binary_accuracy: 0.9462 - loss: 0.1307 - val_binary_accuracy: 0.9546 - val_loss: 0.1368 Epoch 14/15 121/121 ━━━━━━━━━━━━━━━━━━━━ 20s 162ms/step - binary_accuracy: 0.9688 - loss: 0.1086 - val_binary_accuracy: 0.9546 - val_loss: 0.1377 Epoch 15/15 121/121 ━━━━━━━━━━━━━━━━━━━━ 0s 535ms/step - binary_accuracy: 0.9529 - loss: 0.1303
WARNING:absl:You are saving your model as an HDF5 file via `model.save()` or `keras.saving.save_model(model)`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')` or `keras.saving.save_model(model, 'my_model.keras')`.
121/121 ━━━━━━━━━━━━━━━━━━━━ 85s 703ms/step - binary_accuracy: 0.9529 - loss: 0.1302 - val_binary_accuracy: 0.9539 - val_loss: 0.1293
plot_loss_acc(history_resnet_v2, 'ResNet V2')
score = model_resnet_v2.evaluate(ds_test, steps = len(df_test), verbose = 0)
print('Test loss:', score[0])
print('Test accuracy:', score[1])
Test loss: 0.4171968698501587 Test accuracy: 0.8525640964508057
evaluate_model(model_resnet_v2, 'ResNet V2')
Model: ResNet V2 624/624 ━━━━━━━━━━━━━━━━━━━━ 34s 46ms/step Accuracy: 0.8526 Precision: 0.8143 Sensitivity (Recall): 0.9897 Specificity: 0.6239 F1 Score: 0.8935
VGG Model¶
# VGG16 model
base_model_vgg = tf.keras.applications.VGG16(
include_top=False,
weights='imagenet',
input_shape=(IMG_SIZE, IMG_SIZE, 3)
)
base_model_vgg.trainable = True
# Freeze all layers except last 5
for layer in base_model_vgg.layers[:-5]:
layer.trainable = False
def VGGModel():
#Input shape = [width, height, color channels]
inputs = layers.Input(shape=(IMG_SIZE, IMG_SIZE, 3))
x = base_model_vgg(inputs)
# Head
x = layers.GlobalAveragePooling2D()(x)
x = layers.Dense(128, activation='relu')(x)
x = layers.Dropout(0.1)(x)
#Final Layer (Output)
output = layers.Dense(1, activation='sigmoid')(x)
model = keras.Model(inputs=[inputs], outputs=output)
return model
keras.backend.clear_session()
model_vgg = VGGModel()
model_vgg.compile(
loss='binary_crossentropy',
optimizer = keras.optimizers.Adam(learning_rate=1e-5),
metrics=['binary_accuracy']
)
model_vgg.summary()
Model: "functional"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩ │ input_layer (InputLayer) │ (None, 224, 224, 3) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ vgg16 (Functional) │ (None, 7, 7, 512) │ 14,714,688 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ global_average_pooling2d │ (None, 512) │ 0 │ │ (GlobalAveragePooling2D) │ │ │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense (Dense) │ (None, 128) │ 65,664 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dropout (Dropout) │ (None, 128) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_1 (Dense) │ (None, 1) │ 129 │ └─────────────────────────────────┴────────────────────────┴───────────────┘
Total params: 14,780,481 (56.38 MB)
Trainable params: 7,145,217 (27.26 MB)
Non-trainable params: 7,635,264 (29.13 MB)
checkpoint_vgg = callbacks.ModelCheckpoint(
"xray_vgg_model.h5",
save_best_only=True
)
vgg_epoch = 15
history_vgg = model_vgg.fit(
ds_train,
batch_size = BATCH, epochs = vgg_epoch,
validation_data=ds_val,
callbacks=[checkpoint_vgg, early_stopping],
steps_per_epoch=(len(train_df)//BATCH),
validation_steps=(len(val_df)//BATCH)
)
Epoch 1/15 121/121 ━━━━━━━━━━━━━━━━━━━━ 0s 442ms/step - binary_accuracy: 0.8136 - loss: 0.4388
WARNING:absl:You are saving your model as an HDF5 file via `model.save()` or `keras.saving.save_model(model)`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')` or `keras.saving.save_model(model, 'my_model.keras')`.
121/121 ━━━━━━━━━━━━━━━━━━━━ 69s 562ms/step - binary_accuracy: 0.8140 - loss: 0.4379 - val_binary_accuracy: 0.9182 - val_loss: 0.2295 Epoch 2/15 1/121 ━━━━━━━━━━━━━━━━━━━━ 53s 445ms/step - binary_accuracy: 0.9375 - loss: 0.1686
WARNING:absl:You are saving your model as an HDF5 file via `model.save()` or `keras.saving.save_model(model)`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')` or `keras.saving.save_model(model, 'my_model.keras')`.
121/121 ━━━━━━━━━━━━━━━━━━━━ 14s 115ms/step - binary_accuracy: 0.9375 - loss: 0.1686 - val_binary_accuracy: 0.9211 - val_loss: 0.2241 Epoch 3/15 121/121 ━━━━━━━━━━━━━━━━━━━━ 0s 423ms/step - binary_accuracy: 0.9216 - loss: 0.1864
WARNING:absl:You are saving your model as an HDF5 file via `model.save()` or `keras.saving.save_model(model)`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')` or `keras.saving.save_model(model, 'my_model.keras')`.
121/121 ━━━━━━━━━━━━━━━━━━━━ 65s 539ms/step - binary_accuracy: 0.9217 - loss: 0.1863 - val_binary_accuracy: 0.9442 - val_loss: 0.1626 Epoch 4/15 121/121 ━━━━━━━━━━━━━━━━━━━━ 14s 113ms/step - binary_accuracy: 0.9688 - loss: 0.0629 - val_binary_accuracy: 0.9449 - val_loss: 0.1634 Epoch 5/15 121/121 ━━━━━━━━━━━━━━━━━━━━ 0s 421ms/step - binary_accuracy: 0.9324 - loss: 0.1539
WARNING:absl:You are saving your model as an HDF5 file via `model.save()` or `keras.saving.save_model(model)`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')` or `keras.saving.save_model(model, 'my_model.keras')`.
121/121 ━━━━━━━━━━━━━━━━━━━━ 65s 535ms/step - binary_accuracy: 0.9324 - loss: 0.1539 - val_binary_accuracy: 0.9457 - val_loss: 0.1450 Epoch 6/15 121/121 ━━━━━━━━━━━━━━━━━━━━ 14s 115ms/step - binary_accuracy: 0.9688 - loss: 0.1061 - val_binary_accuracy: 0.9457 - val_loss: 0.1465 Epoch 7/15 121/121 ━━━━━━━━━━━━━━━━━━━━ 0s 422ms/step - binary_accuracy: 0.9572 - loss: 0.1222
WARNING:absl:You are saving your model as an HDF5 file via `model.save()` or `keras.saving.save_model(model)`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')` or `keras.saving.save_model(model, 'my_model.keras')`.
121/121 ━━━━━━━━━━━━━━━━━━━━ 66s 544ms/step - binary_accuracy: 0.9572 - loss: 0.1221 - val_binary_accuracy: 0.9516 - val_loss: 0.1210 Epoch 8/15 1/121 ━━━━━━━━━━━━━━━━━━━━ 49s 413ms/step - binary_accuracy: 1.0000 - loss: 0.0257
WARNING:absl:You are saving your model as an HDF5 file via `model.save()` or `keras.saving.save_model(model)`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')` or `keras.saving.save_model(model, 'my_model.keras')`.
121/121 ━━━━━━━━━━━━━━━━━━━━ 14s 113ms/step - binary_accuracy: 1.0000 - loss: 0.0257 - val_binary_accuracy: 0.9546 - val_loss: 0.1197 Epoch 9/15 121/121 ━━━━━━━━━━━━━━━━━━━━ 65s 535ms/step - binary_accuracy: 0.9527 - loss: 0.1101 - val_binary_accuracy: 0.9368 - val_loss: 0.1534 Epoch 10/15 121/121 ━━━━━━━━━━━━━━━━━━━━ 14s 113ms/step - binary_accuracy: 1.0000 - loss: 0.0071 - val_binary_accuracy: 0.9405 - val_loss: 0.1412 Epoch 11/15 121/121 ━━━━━━━━━━━━━━━━━━━━ 64s 534ms/step - binary_accuracy: 0.9627 - loss: 0.1090 - val_binary_accuracy: 0.9427 - val_loss: 0.1355 Epoch 12/15 121/121 ━━━━━━━━━━━━━━━━━━━━ 14s 115ms/step - binary_accuracy: 0.8750 - loss: 0.2185 - val_binary_accuracy: 0.9382 - val_loss: 0.1484 Epoch 13/15 121/121 ━━━━━━━━━━━━━━━━━━━━ 0s 427ms/step - binary_accuracy: 0.9669 - loss: 0.0961
WARNING:absl:You are saving your model as an HDF5 file via `model.save()` or `keras.saving.save_model(model)`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')` or `keras.saving.save_model(model, 'my_model.keras')`.
121/121 ━━━━━━━━━━━━━━━━━━━━ 66s 544ms/step - binary_accuracy: 0.9669 - loss: 0.0961 - val_binary_accuracy: 0.9561 - val_loss: 0.1084 Epoch 14/15 1/121 ━━━━━━━━━━━━━━━━━━━━ 51s 433ms/step - binary_accuracy: 1.0000 - loss: 0.0310
WARNING:absl:You are saving your model as an HDF5 file via `model.save()` or `keras.saving.save_model(model)`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')` or `keras.saving.save_model(model, 'my_model.keras')`.
121/121 ━━━━━━━━━━━━━━━━━━━━ 14s 114ms/step - binary_accuracy: 1.0000 - loss: 0.0310 - val_binary_accuracy: 0.9613 - val_loss: 0.1064 Epoch 15/15 121/121 ━━━━━━━━━━━━━━━━━━━━ 65s 534ms/step - binary_accuracy: 0.9575 - loss: 0.1036 - val_binary_accuracy: 0.9606 - val_loss: 0.1132
plot_loss_acc(history_vgg, 'VGG')
score = model_vgg.evaluate(ds_test, steps = len(df_test), verbose = 0)
print('Test loss:', score[0])
print('Test accuracy:', score[1])
Test loss: 0.3317345082759857 Test accuracy: 0.8878205418586731
Performance Metrics¶
evaluate_model(model_vgg, 'VGG')
Model: VGG 624/624 ━━━━━━━━━━━━━━━━━━━━ 7s 11ms/step Accuracy: 0.8878 Precision: 0.8540 Sensitivity (Recall): 0.9897 Specificity: 0.7179 F1 Score: 0.9169
Testing¶
# VGG16 model
base_model_vgg_testing = tf.keras.applications.VGG16(
include_top=False,
weights='imagenet',
input_shape=(IMG_SIZE, IMG_SIZE, 3)
)
base_model_vgg_testing.trainable = True
# Freeze all layers except last 5
for layer in base_model_vgg_testing.layers[:-5]:
layer.trainable = False
def VGGModel_testing():
#Input shape = [width, height, color channels]
inputs = layers.Input(shape=(IMG_SIZE, IMG_SIZE, 3))
x = base_model_vgg_testing.output
# Head
x = layers.GlobalAveragePooling2D()(x)
x = layers.Dense(128, activation='relu')(x)
x = layers.Dropout(0.1)(x)
#Final Layer (Output)
output = layers.Dense(1, activation='sigmoid')(x)
model = keras.Model(inputs=base_model_vgg_testing.input, outputs=output)
return model
keras.backend.clear_session()
model_vgg_testing = VGGModel_testing()
model_vgg_testing.compile(
loss='binary_crossentropy',
optimizer = keras.optimizers.Adam(learning_rate=1e-5),
metrics=['binary_accuracy']
)
model_vgg_testing.summary()
Model: "functional"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩ │ input_layer_3 (InputLayer) │ (None, 224, 224, 3) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ block1_conv1 (Conv2D) │ (None, 224, 224, 64) │ 1,792 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ block1_conv2 (Conv2D) │ (None, 224, 224, 64) │ 36,928 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ block1_pool (MaxPooling2D) │ (None, 112, 112, 64) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ block2_conv1 (Conv2D) │ (None, 112, 112, 128) │ 73,856 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ block2_conv2 (Conv2D) │ (None, 112, 112, 128) │ 147,584 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ block2_pool (MaxPooling2D) │ (None, 56, 56, 128) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ block3_conv1 (Conv2D) │ (None, 56, 56, 256) │ 295,168 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ block3_conv2 (Conv2D) │ (None, 56, 56, 256) │ 590,080 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ block3_conv3 (Conv2D) │ (None, 56, 56, 256) │ 590,080 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ block3_pool (MaxPooling2D) │ (None, 28, 28, 256) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ block4_conv1 (Conv2D) │ (None, 28, 28, 512) │ 1,180,160 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ block4_conv2 (Conv2D) │ (None, 28, 28, 512) │ 2,359,808 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ block4_conv3 (Conv2D) │ (None, 28, 28, 512) │ 2,359,808 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ block4_pool (MaxPooling2D) │ (None, 14, 14, 512) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ block5_conv1 (Conv2D) │ (None, 14, 14, 512) │ 2,359,808 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ block5_conv2 (Conv2D) │ (None, 14, 14, 512) │ 2,359,808 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ block5_conv3 (Conv2D) │ (None, 14, 14, 512) │ 2,359,808 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ block5_pool (MaxPooling2D) │ (None, 7, 7, 512) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ global_average_pooling2d │ (None, 512) │ 0 │ │ (GlobalAveragePooling2D) │ │ │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense (Dense) │ (None, 128) │ 65,664 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dropout (Dropout) │ (None, 128) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_1 (Dense) │ (None, 1) │ 129 │ └─────────────────────────────────┴────────────────────────┴───────────────┘
Total params: 14,780,481 (56.38 MB)
Trainable params: 7,145,217 (27.26 MB)
Non-trainable params: 7,635,264 (29.13 MB)
checkpoint_vgg_testing = callbacks.ModelCheckpoint(
"xray_vgg_model_testing.h5",
save_best_only=True
)
vgg_epoch = 15
history_vgg_testing = model_vgg_testing.fit(
ds_train,
batch_size = BATCH, epochs = vgg_epoch,
validation_data=ds_val,
callbacks=[checkpoint_vgg_testing, early_stopping],
steps_per_epoch=(len(train_df)//BATCH),
validation_steps=(len(val_df)//BATCH)
)
Epoch 1/15 121/121 ━━━━━━━━━━━━━━━━━━━━ 0s 431ms/step - binary_accuracy: 0.7878 - loss: 0.4629
WARNING:absl:You are saving your model as an HDF5 file via `model.save()` or `keras.saving.save_model(model)`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')` or `keras.saving.save_model(model, 'my_model.keras')`.
121/121 ━━━━━━━━━━━━━━━━━━━━ 68s 551ms/step - binary_accuracy: 0.7883 - loss: 0.4619 - val_binary_accuracy: 0.9159 - val_loss: 0.2608 Epoch 2/15 1/121 ━━━━━━━━━━━━━━━━━━━━ 55s 459ms/step - binary_accuracy: 0.8438 - loss: 0.2318
WARNING:absl:You are saving your model as an HDF5 file via `model.save()` or `keras.saving.save_model(model)`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')` or `keras.saving.save_model(model, 'my_model.keras')`.
121/121 ━━━━━━━━━━━━━━━━━━━━ 14s 117ms/step - binary_accuracy: 0.8438 - loss: 0.2318 - val_binary_accuracy: 0.9107 - val_loss: 0.2572 Epoch 3/15 121/121 ━━━━━━━━━━━━━━━━━━━━ 0s 426ms/step - binary_accuracy: 0.9174 - loss: 0.1960
WARNING:absl:You are saving your model as an HDF5 file via `model.save()` or `keras.saving.save_model(model)`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')` or `keras.saving.save_model(model, 'my_model.keras')`.
121/121 ━━━━━━━━━━━━━━━━━━━━ 65s 540ms/step - binary_accuracy: 0.9175 - loss: 0.1959 - val_binary_accuracy: 0.9256 - val_loss: 0.2214 Epoch 4/15 1/121 ━━━━━━━━━━━━━━━━━━━━ 49s 410ms/step - binary_accuracy: 1.0000 - loss: 0.0758
WARNING:absl:You are saving your model as an HDF5 file via `model.save()` or `keras.saving.save_model(model)`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')` or `keras.saving.save_model(model, 'my_model.keras')`.
121/121 ━━━━━━━━━━━━━━━━━━━━ 14s 115ms/step - binary_accuracy: 1.0000 - loss: 0.0758 - val_binary_accuracy: 0.9315 - val_loss: 0.2046 Epoch 5/15 121/121 ━━━━━━━━━━━━━━━━━━━━ 0s 426ms/step - binary_accuracy: 0.9435 - loss: 0.1560
WARNING:absl:You are saving your model as an HDF5 file via `model.save()` or `keras.saving.save_model(model)`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')` or `keras.saving.save_model(model, 'my_model.keras')`.
121/121 ━━━━━━━━━━━━━━━━━━━━ 66s 545ms/step - binary_accuracy: 0.9435 - loss: 0.1559 - val_binary_accuracy: 0.9464 - val_loss: 0.1485 Epoch 6/15 1/121 ━━━━━━━━━━━━━━━━━━━━ 52s 440ms/step - binary_accuracy: 0.9062 - loss: 0.2025
WARNING:absl:You are saving your model as an HDF5 file via `model.save()` or `keras.saving.save_model(model)`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')` or `keras.saving.save_model(model, 'my_model.keras')`.
121/121 ━━━━━━━━━━━━━━━━━━━━ 14s 115ms/step - binary_accuracy: 0.9062 - loss: 0.2025 - val_binary_accuracy: 0.9464 - val_loss: 0.1475 Epoch 7/15 121/121 ━━━━━━━━━━━━━━━━━━━━ 65s 539ms/step - binary_accuracy: 0.9524 - loss: 0.1312 - val_binary_accuracy: 0.9182 - val_loss: 0.1914 Epoch 8/15 121/121 ━━━━━━━━━━━━━━━━━━━━ 14s 116ms/step - binary_accuracy: 0.9375 - loss: 0.3721 - val_binary_accuracy: 0.9271 - val_loss: 0.1690 Epoch 9/15 121/121 ━━━━━━━━━━━━━━━━━━━━ 0s 439ms/step - binary_accuracy: 0.9594 - loss: 0.1117
WARNING:absl:You are saving your model as an HDF5 file via `model.save()` or `keras.saving.save_model(model)`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')` or `keras.saving.save_model(model, 'my_model.keras')`.
121/121 ━━━━━━━━━━━━━━━━━━━━ 67s 556ms/step - binary_accuracy: 0.9595 - loss: 0.1117 - val_binary_accuracy: 0.9546 - val_loss: 0.1113 Epoch 10/15 1/121 ━━━━━━━━━━━━━━━━━━━━ 54s 451ms/step - binary_accuracy: 0.9062 - loss: 0.1948
WARNING:absl:You are saving your model as an HDF5 file via `model.save()` or `keras.saving.save_model(model)`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')` or `keras.saving.save_model(model, 'my_model.keras')`.
121/121 ━━━━━━━━━━━━━━━━━━━━ 14s 117ms/step - binary_accuracy: 0.9062 - loss: 0.1948 - val_binary_accuracy: 0.9613 - val_loss: 0.1092 Epoch 11/15 121/121 ━━━━━━━━━━━━━━━━━━━━ 67s 553ms/step - binary_accuracy: 0.9551 - loss: 0.1178 - val_binary_accuracy: 0.9531 - val_loss: 0.1163 Epoch 12/15 121/121 ━━━━━━━━━━━━━━━━━━━━ 15s 117ms/step - binary_accuracy: 0.9062 - loss: 0.1563 - val_binary_accuracy: 0.9546 - val_loss: 0.1125 Epoch 13/15 121/121 ━━━━━━━━━━━━━━━━━━━━ 66s 548ms/step - binary_accuracy: 0.9641 - loss: 0.0964 - val_binary_accuracy: 0.9531 - val_loss: 0.1202 Epoch 14/15 121/121 ━━━━━━━━━━━━━━━━━━━━ 14s 117ms/step - binary_accuracy: 0.9688 - loss: 0.0745 - val_binary_accuracy: 0.9554 - val_loss: 0.1134 Epoch 15/15 121/121 ━━━━━━━━━━━━━━━━━━━━ 0s 436ms/step - binary_accuracy: 0.9657 - loss: 0.0867
WARNING:absl:You are saving your model as an HDF5 file via `model.save()` or `keras.saving.save_model(model)`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')` or `keras.saving.save_model(model, 'my_model.keras')`.
121/121 ━━━━━━━━━━━━━━━━━━━━ 67s 557ms/step - binary_accuracy: 0.9657 - loss: 0.0867 - val_binary_accuracy: 0.9568 - val_loss: 0.1053
plot_loss_acc(history_vgg_testing, 'VGG Testing')
score = model_vgg_testing.evaluate(ds_test, steps = len(df_test), verbose = 0)
print('Test loss:', score[0])
print('Test accuracy:', score[1])
Test loss: 0.27783361077308655 Test accuracy: 0.9038461446762085
evaluate_model(model_vgg_testing, 'VGG Testing')
Model: VGG Testing 624/624 ━━━━━━━━━━━━━━━━━━━━ 7s 11ms/step Accuracy: 0.9038 Precision: 0.8767 Sensitivity (Recall): 0.9846 Specificity: 0.7692 F1 Score: 0.9275
# Checking all the wrongly predicted images
y_pred_proba = model_vgg_testing.predict(ds_test, steps=len(df_test), verbose=1)
y_true = np.array(ds_test.classes)
if hasattr(ds_test, "filepaths"):
filepaths = np.array(ds_test.filepaths)
elif hasattr(ds_test, "filenames"):
filepaths = np.array(ds_test.filenames)
else:
filepaths = np.array(df_test['image'])
if y_pred_proba.ndim == 2 and y_pred_proba.shape[1] > 1:
y_pred = np.argmax(y_pred_proba, axis=1)
pred_score = y_pred_proba.max(axis=1)
else:
y_pred_proba = y_pred_proba.ravel()
y_pred = (y_pred_proba > 0.5).astype(int)
pred_score = y_pred_proba
results_df = pd.DataFrame({
"filepath": filepaths,
"y_true": y_true,
"y_pred": y_pred,
"pred_score": pred_score
})
wrong_df = results_df[results_df["y_true"] != results_df["y_pred"]].reset_index(drop=True)
print(f"Total samples: {len(results_df)}")
print(f"Wrongly predicted: {len(wrong_df)}")
624/624 ━━━━━━━━━━━━━━━━━━━━ 7s 12ms/step Total samples: 624 Wrongly predicted: 60
wrong_df.sort_values(['filepath'])
| filepath | y_true | y_pred | pred_score | |
|---|---|---|---|---|
| 19 | ../Data/chest_xray/test/NORMAL/IM-0022-0001.jpeg | 0 | 1 | 0.989724 |
| 20 | ../Data/chest_xray/test/NORMAL/IM-0036-0001.jpeg | 0 | 1 | 0.705035 |
| 4 | ../Data/chest_xray/test/NORMAL/IM-0073-0001.jpeg | 0 | 1 | 0.975362 |
| 17 | ../Data/chest_xray/test/NORMAL/NORMAL2-IM-0007-0001.jpeg | 0 | 1 | 0.664389 |
| 27 | ../Data/chest_xray/test/NORMAL/NORMAL2-IM-0028-0001.jpeg | 0 | 1 | 0.871873 |
| 41 | ../Data/chest_xray/test/NORMAL/NORMAL2-IM-0030-0001.jpeg | 0 | 1 | 0.636974 |
| 26 | ../Data/chest_xray/test/NORMAL/NORMAL2-IM-0052-0001.jpeg | 0 | 1 | 0.856610 |
| 31 | ../Data/chest_xray/test/NORMAL/NORMAL2-IM-0059-0001.jpeg | 0 | 1 | 0.790313 |
| 38 | ../Data/chest_xray/test/NORMAL/NORMAL2-IM-0066-0001.jpeg | 0 | 1 | 0.713210 |
| 25 | ../Data/chest_xray/test/NORMAL/NORMAL2-IM-0073-0001.jpeg | 0 | 1 | 0.803488 |
| 13 | ../Data/chest_xray/test/NORMAL/NORMAL2-IM-0081-0001.jpeg | 0 | 1 | 0.600887 |
| 9 | ../Data/chest_xray/test/NORMAL/NORMAL2-IM-0086-0001.jpeg | 0 | 1 | 0.881516 |
| 15 | ../Data/chest_xray/test/NORMAL/NORMAL2-IM-0095-0001.jpeg | 0 | 1 | 0.504454 |
| 29 | ../Data/chest_xray/test/NORMAL/NORMAL2-IM-0096-0001.jpeg | 0 | 1 | 0.504454 |
| 0 | ../Data/chest_xray/test/NORMAL/NORMAL2-IM-0102-0001.jpeg | 0 | 1 | 0.870772 |
| 18 | ../Data/chest_xray/test/NORMAL/NORMAL2-IM-0105-0001.jpeg | 0 | 1 | 0.607571 |
| 42 | ../Data/chest_xray/test/NORMAL/NORMAL2-IM-0107-0001.jpeg | 0 | 1 | 0.576215 |
| 21 | ../Data/chest_xray/test/NORMAL/NORMAL2-IM-0111-0001.jpeg | 0 | 1 | 0.723050 |
| 37 | ../Data/chest_xray/test/NORMAL/NORMAL2-IM-0120-0001.jpeg | 0 | 1 | 0.664849 |
| 46 | ../Data/chest_xray/test/NORMAL/NORMAL2-IM-0132-0001.jpeg | 0 | 1 | 0.910103 |
| 50 | ../Data/chest_xray/test/NORMAL/NORMAL2-IM-0145-0001.jpeg | 0 | 1 | 0.772149 |
| 34 | ../Data/chest_xray/test/NORMAL/NORMAL2-IM-0150-0001.jpeg | 0 | 1 | 0.517558 |
| 32 | ../Data/chest_xray/test/NORMAL/NORMAL2-IM-0171-0001.jpeg | 0 | 1 | 0.829799 |
| 44 | ../Data/chest_xray/test/NORMAL/NORMAL2-IM-0173-0001-0001.jpeg | 0 | 1 | 0.677606 |
| 3 | ../Data/chest_xray/test/NORMAL/NORMAL2-IM-0173-0001-0002.jpeg | 0 | 1 | 0.677606 |
| 48 | ../Data/chest_xray/test/NORMAL/NORMAL2-IM-0195-0001.jpeg | 0 | 1 | 0.947088 |
| 6 | ../Data/chest_xray/test/NORMAL/NORMAL2-IM-0196-0001.jpeg | 0 | 1 | 0.957785 |
| 53 | ../Data/chest_xray/test/NORMAL/NORMAL2-IM-0198-0001.jpeg | 0 | 1 | 0.947972 |
| 35 | ../Data/chest_xray/test/NORMAL/NORMAL2-IM-0199-0001.jpeg | 0 | 1 | 0.924133 |
| 33 | ../Data/chest_xray/test/NORMAL/NORMAL2-IM-0201-0001.jpeg | 0 | 1 | 0.911743 |
| 40 | ../Data/chest_xray/test/NORMAL/NORMAL2-IM-0206-0001.jpeg | 0 | 1 | 0.995018 |
| 16 | ../Data/chest_xray/test/NORMAL/NORMAL2-IM-0210-0001.jpeg | 0 | 1 | 0.990126 |
| 43 | ../Data/chest_xray/test/NORMAL/NORMAL2-IM-0219-0001.jpeg | 0 | 1 | 0.998575 |
| 52 | ../Data/chest_xray/test/NORMAL/NORMAL2-IM-0221-0001.jpeg | 0 | 1 | 0.977822 |
| 8 | ../Data/chest_xray/test/NORMAL/NORMAL2-IM-0222-0001.jpeg | 0 | 1 | 0.986483 |
| 1 | ../Data/chest_xray/test/NORMAL/NORMAL2-IM-0229-0001.jpeg | 0 | 1 | 0.962761 |
| 28 | ../Data/chest_xray/test/NORMAL/NORMAL2-IM-0232-0001.jpeg | 0 | 1 | 0.990768 |
| 39 | ../Data/chest_xray/test/NORMAL/NORMAL2-IM-0233-0001.jpeg | 0 | 1 | 0.990765 |
| 22 | ../Data/chest_xray/test/NORMAL/NORMAL2-IM-0237-0001.jpeg | 0 | 1 | 0.969171 |
| 45 | ../Data/chest_xray/test/NORMAL/NORMAL2-IM-0238-0001.jpeg | 0 | 1 | 0.981214 |
| 7 | ../Data/chest_xray/test/NORMAL/NORMAL2-IM-0241-0001.jpeg | 0 | 1 | 0.955723 |
| 51 | ../Data/chest_xray/test/NORMAL/NORMAL2-IM-0246-0001-0001.jpeg | 0 | 1 | 0.533083 |
| 2 | ../Data/chest_xray/test/NORMAL/NORMAL2-IM-0246-0001-0002.jpeg | 0 | 1 | 0.533083 |
| 11 | ../Data/chest_xray/test/NORMAL/NORMAL2-IM-0246-0001.jpeg | 0 | 1 | 0.998966 |
| 24 | ../Data/chest_xray/test/NORMAL/NORMAL2-IM-0251-0001.jpeg | 0 | 1 | 0.557734 |
| 12 | ../Data/chest_xray/test/NORMAL/NORMAL2-IM-0252-0001.jpeg | 0 | 1 | 0.991211 |
| 47 | ../Data/chest_xray/test/NORMAL/NORMAL2-IM-0256-0001.jpeg | 0 | 1 | 0.966066 |
| 23 | ../Data/chest_xray/test/NORMAL/NORMAL2-IM-0259-0001.jpeg | 0 | 1 | 0.906016 |
| 10 | ../Data/chest_xray/test/NORMAL/NORMAL2-IM-0267-0001.jpeg | 0 | 1 | 0.972706 |
| 36 | ../Data/chest_xray/test/NORMAL/NORMAL2-IM-0316-0001.jpeg | 0 | 1 | 0.865446 |
| 30 | ../Data/chest_xray/test/NORMAL/NORMAL2-IM-0330-0001.jpeg | 0 | 1 | 0.831004 |
| 14 | ../Data/chest_xray/test/NORMAL/NORMAL2-IM-0338-0001.jpeg | 0 | 1 | 0.535172 |
| 49 | ../Data/chest_xray/test/NORMAL/NORMAL2-IM-0373-0001.jpeg | 0 | 1 | 0.873923 |
| 5 | ../Data/chest_xray/test/NORMAL/NORMAL2-IM-0376-0001.jpeg | 0 | 1 | 0.924931 |
| 54 | ../Data/chest_xray/test/PNEUMONIA/person119_bacteria_568.jpeg | 1 | 0 | 0.410172 |
| 55 | ../Data/chest_xray/test/PNEUMONIA/person152_bacteria_724.jpeg | 1 | 0 | 0.313180 |
| 58 | ../Data/chest_xray/test/PNEUMONIA/person153_bacteria_726.jpeg | 1 | 0 | 0.026573 |
| 56 | ../Data/chest_xray/test/PNEUMONIA/person154_bacteria_728.jpeg | 1 | 0 | 0.009726 |
| 59 | ../Data/chest_xray/test/PNEUMONIA/person16_virus_47.jpeg | 1 | 0 | 0.127890 |
| 57 | ../Data/chest_xray/test/PNEUMONIA/person173_bacteria_831.jpeg | 1 | 0 | 0.405089 |
def predict_single_image(img_path, model, img_size=224, class_names=['Normal', 'Pneumonia'], preprocess_fn=None):
orig_bgr = cv2.imread(img_path)
if orig_bgr is None:
raise ValueError(f"Could not read image at {img_path}")
orig_bgr = cv2.resize(orig_bgr, (img_size, img_size))
orig_rgb = cv2.cvtColor(orig_bgr, cv2.COLOR_BGR2RGB)
# produce array for model input
img_arr = orig_rgb.astype("float32")
img_batch = np.expand_dims(img_arr, axis=0) # (1,H,W,3)
if preprocess_fn is not None:
img_for_model = preprocess_fn(np.copy(img_batch))
else:
img_for_model = img_batch / 255.0
# Predict
preds = model.predict(img_for_model, verbose=0)
if preds[0] > 0.5:
pred_class_idx = 1
else:
pred_class_idx = 0
if class_names:
pred_class_name = class_names[pred_class_idx]
else:
pred_class_name = str(pred_class_idx)
return {
"prediction_probability": preds[0],
"predicted_class": pred_class_name
}
img_path = '../Data/chest_xray/test/NORMAL/IM-0005-0001.jpeg'
result = predict_single_image(img_path, model_vgg_testing, img_size=224)
print(result)
{'prediction_probability': array([0.02490874], dtype=float32), 'predicted_class': 'Normal'}
img_path = '../Data/chest_xray/test/PNEUMONIA/person3_virus_17.jpeg'
result = predict_single_image(img_path, model_vgg_testing, img_size=224)
print(result)
{'prediction_probability': array([0.6375263], dtype=float32), 'predicted_class': 'Pneumonia'}
Explainability¶
Performance metrics: Accuracy, Precision, Recall, F1 score, ROC-AUC score
| Model | Accuracy | Precision | Recall | F1 Score | ROC-AUC |
|---|---|---|---|---|---|
| CNN | 0.8029 | 0.7811 | 0.9513 | 0.8578 | 0.9144 |
| ResNet | 0.7837 | 0.7576 | 0.9615 | 0.8475 | 0.8917 |
| ResNetV2 | 0.8526 | 0.8143 | 0.9897 | 0.8935 | 0.9467 |
| VGG | 0.8878 | 0.8540 | 0.9897 | 0.9169 | 0.9665 |
| VGG Test | 0.9038 | 0.8767 | 0.9846 | 0.9275 | 0.9683 |
From the above table, we can infer that VGG Test model performed the best out of all the model. Moving forward, we will take VGG as the main model.
An important part of the analysis is model explainability using Grad-CAM (Gradient-weighted Class Activation Mapping). This technique generates a heatmap overlay on the original image, highlighting the specific regions that the model focused on to make its prediction
img_path_pneumonia = '../Data/chest_xray/test/PNEUMONIA/person1_virus_13.jpeg'
img_path_normal = '../Data/chest_xray/test/NORMAL/IM-0006-0001.jpeg'
model_vgg_testing.summary()
Model: "functional"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩ │ input_layer_3 (InputLayer) │ (None, 224, 224, 3) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ block1_conv1 (Conv2D) │ (None, 224, 224, 64) │ 1,792 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ block1_conv2 (Conv2D) │ (None, 224, 224, 64) │ 36,928 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ block1_pool (MaxPooling2D) │ (None, 112, 112, 64) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ block2_conv1 (Conv2D) │ (None, 112, 112, 128) │ 73,856 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ block2_conv2 (Conv2D) │ (None, 112, 112, 128) │ 147,584 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ block2_pool (MaxPooling2D) │ (None, 56, 56, 128) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ block3_conv1 (Conv2D) │ (None, 56, 56, 256) │ 295,168 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ block3_conv2 (Conv2D) │ (None, 56, 56, 256) │ 590,080 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ block3_conv3 (Conv2D) │ (None, 56, 56, 256) │ 590,080 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ block3_pool (MaxPooling2D) │ (None, 28, 28, 256) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ block4_conv1 (Conv2D) │ (None, 28, 28, 512) │ 1,180,160 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ block4_conv2 (Conv2D) │ (None, 28, 28, 512) │ 2,359,808 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ block4_conv3 (Conv2D) │ (None, 28, 28, 512) │ 2,359,808 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ block4_pool (MaxPooling2D) │ (None, 14, 14, 512) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ block5_conv1 (Conv2D) │ (None, 14, 14, 512) │ 2,359,808 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ block5_conv2 (Conv2D) │ (None, 14, 14, 512) │ 2,359,808 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ block5_conv3 (Conv2D) │ (None, 14, 14, 512) │ 2,359,808 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ block5_pool (MaxPooling2D) │ (None, 7, 7, 512) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ global_average_pooling2d │ (None, 512) │ 0 │ │ (GlobalAveragePooling2D) │ │ │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense (Dense) │ (None, 128) │ 65,664 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dropout (Dropout) │ (None, 128) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_1 (Dense) │ (None, 1) │ 129 │ └─────────────────────────────────┴────────────────────────┴───────────────┘
Total params: 29,070,917 (110.90 MB)
Trainable params: 7,145,217 (27.26 MB)
Non-trainable params: 7,635,264 (29.13 MB)
Optimizer params: 14,290,436 (54.51 MB)
pretrained_model = model_vgg_testing
def grad_cam_evaluateV2(
img_path,
model,
classifier_layer_names=['global_average_pooling2d', 'dense', 'dropout', 'dense_1'],
classifier_input_shape=(7, 7, 512),
last_conv_layer_name="block5_pool",
preprocess_fn=lambda x: x/255.0,
img_size=IMG_SIZE,
alpha=0.4,
class_names=['Normal', 'Pneumonia']
):
# --- load + prepare image ---
orig_bgr = cv2.imread(img_path)
if orig_bgr is None:
raise ValueError(f"Could not read image at {img_path}")
orig_bgr = cv2.resize(orig_bgr, (img_size, img_size))
orig_rgb = cv2.cvtColor(orig_bgr, cv2.COLOR_BGR2RGB)
# produce array for model input
img_arr = orig_rgb.astype("float32")
img_batch = np.expand_dims(img_arr, axis=0) # (1,H,W,3)
if preprocess_fn is not None:
img_for_model = preprocess_fn(np.copy(img_batch))
else:
img_for_model = img_batch / 255.0
# --- get predictions ---
preds = model.predict(img_for_model, verbose=0)
# --- construct last-conv model and small classifier that maps conv output -> preds ---
try:
last_conv_layer = model.get_layer(last_conv_layer_name)
except Exception as e:
raise ValueError(f"Layer '{last_conv_layer_name}' not found. Available layers: {[l.name for l in model.layers[-10:]]}") from e
last_conv_layer_model = Model(model.input, last_conv_layer.output)
# build classifier (taking conv feature map as input)
classifier_input = Input(shape=classifier_input_shape)
x = classifier_input
for layer_name in classifier_layer_names:
layer = model.get_layer(layer_name)
# re-apply the same layer on x
x = layer(x)
classifier_model = Model(classifier_input, x)
# --- compute gradients and heatmap ---
with tf.GradientTape() as tape:
# ensure we evaluate with training=False (BatchNorm behavior)
conv_output = last_conv_layer_model(img_for_model, training=False) # (1,H,W,C)
preds_from_conv = classifier_model(conv_output, training=False)
print('preds_from_conv: ',preds_from_conv)
# choose top predicted class (works for binary or multi-class)
pred_from_conv_numpy = preds_from_conv.numpy()
print(pred_from_conv_numpy)
if preds.ravel() < 0.5:
top_index = 0
else:
top_index = 1
top_class_channel = preds_from_conv[:, 0]
class_predicted = class_names[top_index]
grads = tape.gradient(top_class_channel, conv_output) # (1,H,W,C)
pooled_grads = tf.reduce_mean(grads, axis=(0, 1, 2)).numpy() # (C,)
conv_output_np = conv_output.numpy()[0] # (H,W,C)
# weight channels by importance
for i in range(pooled_grads.shape[-1]):
conv_output_np[:, :, i] *= pooled_grads[i]
heatmap = np.mean(conv_output_np, axis=-1) # (H,W)
heatmap = np.maximum(heatmap, 0)
max_val = np.max(heatmap) if np.max(heatmap) != 0 else 1e-10
heatmap = heatmap / max_val # normalized [0,1]
# --- create overlay ---
heatmap_resized = cv2.resize(heatmap, (img_size, img_size))
heatmap_uint8 = np.uint8(255 * heatmap_resized)
heatmap_color = cv2.applyColorMap(heatmap_uint8, cv2.COLORMAP_JET)
heatmap_color = cv2.cvtColor(heatmap_color, cv2.COLOR_BGR2RGB)
overlay = cv2.addWeighted(orig_rgb.astype('uint8'), 1 - alpha, heatmap_color, alpha, 0)
# --- plotting ---
plt.figure(figsize=(10,4))
plt.subplot(1,3,1)
plt.imshow(orig_rgb.astype('uint8'))
plt.title("Original")
plt.axis('off')
plt.subplot(1,3,2)
plt.imshow(heatmap_resized, cmap='jet')
plt.title("Grad-CAM heatmap")
plt.axis('off')
plt.subplot(1,3,3)
plt.imshow(overlay)
plt.title("Overlay")
plt.axis('off')
plt.suptitle(f"Prediction: {preds.ravel()} Prediction: {class_predicted}", fontsize=12)
plt.tight_layout()
plt.show()
grad_cam_evaluateV2(
img_path=img_path_pneumonia,
model=pretrained_model
)
preds_from_conv: tf.Tensor([[0.9877753]], shape=(1, 1), dtype=float32) [[0.9877753]]
grad_cam_evaluateV2(
img_path='../Data/chest_xray/test/PNEUMONIA/person3_virus_17.jpeg',
model=pretrained_model
)
preds_from_conv: tf.Tensor([[0.47859532]], shape=(1, 1), dtype=float32) [[0.47859532]]
grad_cam_evaluateV2(
img_path='../Data/chest_xray/test/NORMAL/IM-0025-0001.jpeg',
model=pretrained_model
)
preds_from_conv: tf.Tensor([[0.01514762]], shape=(1, 1), dtype=float32) [[0.01514762]]
The Grad-CAM visualizations showed that the models were correctly focusing on key areas of the lungs, building confidence in their diagnostic decisions.
Results and Discussion¶
The transfer learning models, particularly the fine-tuned versions, significantly outperformed the custom CNN. This highlights the effectiveness of using pre-trained models on smaller, specialized datasets. The models achieved high accuracy, with an emphasis on recall for the pneumonia class, which is a critical metric in a medical context to minimize false negatives (failing to detect a case of pneumonia).
Potential of the Model¶
Triage & prioritization
- Flag high-risk X-rays so urgent cases are read earlier (reduce time-to-diagnosis).
- Useful in high workload settings or EDs where rapid triage matters.
Second-reader / decision support
- Provide a second opinion (probability + heatmap) to increase radiologist sensitivity for subtle pneumonia.
- Helps less-experienced readers, e.g., in smaller hospitals or tele-radiology.
Quantification & monitoring
- Track lesion probability and heatmap changes across serial studies to help assess progression or response to treatment.
Workflow automation
- Auto-populate structured reporting fields (e.g., “probability of pneumonia: 0.87”) to reduce clerical load.
Data capture for research
- Automatically tag studies for cohort creation (for clinical studies or QA).
Limitations & realistic expectations
- Not a replacement for radiologist judgement — should be human-in-the-loop.
- Performance may degrade on images from unseen hospitals, different X-ray machines, or different patient populations.
- Susceptible to dataset bias (age, positioning, comorbidities, portable AP vs PA view).
- Explainability (Grad-CAM) helps but is not a proof of correctness.
Steps for Clinical Integration¶
1. Validation¶
- Collect diverse, labeled X-rays from multiple centers.
- Test on both internal and external data.
- Report AUC, sensitivity, specificity, PPV/NPV, calibration, and subgroup analysis.
2. Technical Setup¶
- Package model (SavedModel/ONNX).
- Provide inference API (REST/gRPC) for DICOM input, probability, and Grad-CAM overlay.
- Integrate with PACS/RIS, ensuring low latency (<3s).
3. Workflow Integration¶
- Start with silent mode (predictions not shown).
- Move to pilot phase where radiologists see results as optional overlays.
- Provide clear UI (probability + heatmap) and short training for clinicians.
4. Clinical Use & Monitoring¶
- Deploy gradually (single unit → multi-center).
- Track accuracy, bias across subgroups, and radiologist feedback.
- Recalibrate/retrain with new local data.
- Ensure compliance (HIPAA/GDPR, FDA/CE if needed).
The model can speed up pneumonia detection and support radiologists, but it needs staged validation, safe workflow integration, and continuous monitoring to be clinically useful.
Conclusions¶
The project successfully demonstrated that deep learning models, especially those utilizing transfer learning, can accurately classify chest X-ray images for pneumonia detection. The high performance and ability to provide visual explanations (via Grad-CAM) suggest that these models have significant potential as a diagnostic aid.
Future Work¶
For future work and clinical integration, several steps are necessary:
Clinical Validation
The model needs to be validated on a larger, more diverse dataset from multiple hospitals and imaging centers. This multi-center validation is crucial to ensure the model generalizes well across different patient demographics (e.g., age, race, gender) and types of imaging equipment. This step will help identify and mitigate potential algorithmic bias before clinical use.
Workflow Integration
The model should be integrated into existing clinical systems, such as a Picture Archiving and Communication System (PACS) or a Radiology Information System (RIS). This could involve providing a preliminary triage classification for radiologists, or a second-opinion overlay on the image itself. The system should be designed to be non-intrusive and to provide results with low latency, allowing clinicians to receive predictions in near real-time.
Continuous Monitoring
Once deployed, the model's performance needs to be continuously monitored in a real-world setting. A robust MLOps pipeline is required to track model performance metrics, detect data drift (changes in the data distribution over time), and identify concept drift (changes in the relationship between input data and the target variable). This monitoring will inform when the model needs to be retrained on new, incoming data.
Improved Data and Feature Engineering
The model's diagnostic accuracy can be further improved by incorporating additional clinical data. While the current model relies solely on images, a more comprehensive system could benefit from integrating structured patient data (e.g., patient demographics, lab results, vital signs, comorbidities) and unstructured data from clinical notes (using natural language processing). A multimodal approach could provide a more holistic view and lead to a more robust prediction.
Enhanced Explainability
While Grad-CAM provides a good starting point, future work should explore more advanced explainability techniques. Providing clinicians with clear and intuitive explanations for the model's predictions is critical for building trust and ensuring the model is used appropriately. Techniques like LIME or SHAP could be used to explain predictions at a per-patient level.
Ethical and Regulatory Compliance
Moving toward a clinically viable tool requires addressing ethical and regulatory considerations. The project would need to seek approval from regulatory bodies (e.g., FDA in the U.S., CE Mark in Europe) for use as a medical device. Additionally, a clear framework for data privacy, patient consent, and algorithmic fairness must be established to ensure the tool does not perpetuate existing health disparities.
The application of deep learning to medical imaging holds immense promise for improving healthcare efficiency and accuracy, and this project provides a strong foundation for a clinically viable tool.